Design patterns for ML pipelines have evolved several times in the past decade. These changes are usually driven by imbalances between memory and CPU performance. They are also distinct from traditional data processing pipelines (something like map reduce) as they need to support the execution of long-running, stateful tasks associated with deep learning. As growth in dataset sizes outpace memory availability, we have seen more ETL pipelines designed with distributed training and distributed storage as first-class principles. Not only can these pipelines train models in a parallel fashion using multiple accelerators, but they can also replace traditional distributed file systems with cloud object stores.

Let's say you have 3 GPUs available and you want to train a model on one of them. To allow Pytorch to "see" all available GPUs, use: There are a few different ways to use multiple GPUs, including data parallelism and model parallelism. Data parallelism refers to using multiple GPUs to increase the number of examples processed simultaneously. For example, if a batch size of 256 fits on one GPU, you can use data parallelism to increase the batch size to 512 by using two GPUs, and Pytorch will automatically assign 256 examples to one GPU and 256 examples to the other GPU. Using data parallelism can be accomplished easily through DataParallel.

Deep Learning (the favourite buzzword of late 2010s along with blockchain/bitcoin and Data Science/Machine Learning) has enabled us to do some really cool stuff the last few years. Other than the advances in algorithms (which admittedly are based on ideas already known since 1990s aka "Data Mining era"), the main reasons of its success can be attributed to the availability of large free datasets, the introduction of open-source libraries and the use of GPUs. In this blog post I will focus on the last two and I'll share with you some tips that I learned the hard way. TensorFlow is a very popular Deep Learning library developed by Google which allows you to prototype quickly complex networks. It comes with lots of interesting features such as auto-differentiation (which saves you from estimating/coding the gradients of the cost functions) and GPU support (which allows you to get easily a 200x speed improvement using decent hardware).

At Zalando Research, as in most AI research departments, we realize the importance of experimenting and quickly prototyping ideas. With datasets getting bigger it thus becomes useful to know how to train deep learning models quickly and efficiently on the shared resources we have. TensorFlow's Estimators API is useful for training models in a distributed environment with multiple GPUs. Here, we'll present this workflow by training a custom estimator written with tf.keras for the tiny Fashion-MNIST dataset, and then show a more practical use case at the end. Note: there's also a cool new feature the TensorFlow team has been working on, (which at the time of writing is still in master), that lets you train a tf.keras model without first needing to convert it to an Estimator, with just a couple lines of additional code!

Using Keras to train deep neural networks with multiple GPUs (Photo credit: Nor-Tech.com). Keras is undoubtedly my favorite deep learning Python framework, especially for image classification. I use Keras in production applications, in my personal deep learning projects, and here on the PyImageSearch blog. I've even based over two-thirds of my new book, Deep Learning for Computer Vision with Python on Keras. However, one of my biggest hangups with Keras is that it can be a pain to perform multi-GPU training.

Using Keras to train deep neural networks with multiple GPUs (Photo credit: Nor-Tech.com). Keras is undoubtedly my favorite deep learning Python framework, especially for image classification. I use Keras in production applications, in my personal deep learning projects, and here on the PyImageSearch blog. I've even based over two-thirds of my new book, Deep Learning for Computer Vision with Python on Keras. However, one of my biggest hangups with Keras is that it can be a pain to perform multi-GPU training.